Metadata-preserved audio object clustering

ABSTRACT

Example embodiments disclosed herein relate to audio object clustering. A method for metadata-preserved audio object clustering is disclosed. The method comprises classifying a plurality of audio objects into a number of categories based on information to be preserved in metadata associated with the plurality of audio objects. The method further comprises assigning a predetermined number of clusters to the categories and allocating an audio object in each of the categories to at least one of the clusters according to the assigning. Corresponding system and computer program product are also disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to Chinese PatentApplication No. 201410765578.6 filed 11 Dec. 2014 and U.S. ProvisionalPatent Application No. 62/100,183 filed 6 Jan. 2015, the contents ofeach are hereby incorporated by reference in their entirety.

TECHNOLOGY

Example embodiments disclosed herein generally relate to audio contentprocessing, and more specifically to a method and system for audioobject clustering which allows metadata to be preserved.

BACKGROUND

The advent of object-based audio has significantly increased the amountof audio data and the complexity of rendering this data within high-endplayback systems. For example, cinema sound tracks may comprise manydifferent sound elements corresponding to images on the screen, dialogs,noises, and sound effects that emanate from different places on thescreen and those sound tracks also are combined with background musicand ambient effects to create an overall auditory experience. Accurateplayback requires that the sounds are reproduced in a way thatcorresponds as closely as possible to what is shown on the screen withrespect to the position, intensity, movement, and depth of the soundsource. Object-based audio represents a significant improvement overtraditional channel-based audio systems that send audio content in theform of speaker feeds to individual speakers in a listening environmentand are thus relatively limited with respect to the spatial playback ofspecific audio objects.

The introduction of digital cinema and the development ofthree-dimensional (“3D”) content have created new standards for sound,such as the incorporation of multiple channels of audio to allow greatercreativity for content creators, and a more enveloping and realisticauditory experience for audiences. It is critical to expand beyond thetraditional speaker feeds and channel-based audio as a means fordistributing spatial audio. Moreover, there has been considerableinterest in a model-based audio description that allows a listener toselect a desired playback configuration with the audio renderedspecifically for the chosen configuration. The spatial presentation ofsound utilizes audio objects which are audio signals with associatedparametric source descriptions, such as apparent source position (e.g.,3D coordinates), apparent source width, and other parameters. Furtheradvancements include a next generation spatial audio (also referred toas “adaptive audio”) format that has been developed by including a mixof audio objects and traditional channel-based speaker feeds (audiobeds) along with the positional metadata for the audio objects.

As used herein, the term “audio object” refers to an individual audioelement that exists for a defined duration of time in the sound field.The term “audio bed” or “bed” refers to audio channels that are meant tobe reproduced in predefined and fixed speaker locations.

In some soundtracks, there may be several (e.g., 7, 9, or 11) bedchannels containing audio. Additionally, based on the capabilities of anauthoring system, there may be tens or even hundreds of individual audioobjects that are combined during rendering to create a spatially diverseand immersive audio experience. In other distribution and transmissionsystems, there may be an available bandwidth large enough to transmitall audio beds and objects with little or no audio compression. In somecases, however, such as Blu-ray disc, broadcast (cable, satellite andterrestrial), mobile (3G and 4G) and over-the-top (OTT, or Internet)distribution, there may be significant limitations on the availablebandwidth to digitally transmit all of the bed and object informationcreated at the time of authoring. While audio coding methods (lossy orlossless) may be applied to the audio to reduce the required bandwidth,audio coding may not be sufficient to reduce the bandwidth required totransmit the audio, particularly over very limited networks, such asmobile 3G and 4G networks.

Some prior methods have been developed to reduce the number of objectsinput into a smaller set of output objects by means of clustering.Generally in some clustering processes, metadata such as size, zonemask, and snap should be pre-rendered to an internal channel layout. Theclustering of audio objects is only based on spatial position of audioobjects, and the output objects only contain the positional metadata.This kind of output objects may not work well for some reproducedsystems, since the loss of metadata may violate the expected artisticintent.

The subject matter discussed in the background section should not beassumed as the prior art merely due to its disclosure in the backgroundsection. Similarly, a problem mentioned in the background section orassociated with the subject matter of the background section should notbe assumed to have been previously recognized in the prior art. Thesubject matter in the background section merely represents differentapproaches, which in and of themselves may also be example embodiments.

SUMMARY

In order to address the foregoing and other potential problems, exampleembodiments proposes a method and system for metadata-preserved audioobject clustering.

In one aspect, example embodiments provide a method formetadata-preserved audio object clustering. The method includesclassifying a plurality of audio objects into a number of categoriesbased on information to be preserved in metadata associated with theplurality of audio objects. The method further includes assigning apredetermined number of clusters to the categories and allocating anaudio object in each of the categories to at least one of the clustersaccording to the assigning. Embodiments in this regard further include acorresponding computer program product.

In another aspect, example embodiments provide a system formetadata-preserved audio object clustering. The system includes an audioobject classification unit configured to classify a plurality of audioobjects into a number of categories based on information to be preservedin metadata associated with the plurality of audio objects. The systemfurther includes a cluster assignment unit configured to assign apredetermined number of clusters to the categories and an audio objectallocation unit configured to allocate an audio object in each of thecategories to at least one of the clusters according to the assigning.

Through the following description, it would be appreciated that inaccordance with the example embodiments disclosed herein, input audioobjects are classified into corresponding categories depending on theirinformation to be preserved in metadata, so that different metadata tobe preserved or a unique combination of metadata to be preserved isassociated with a different category. After clustering, for an audioobject within one category, it is less possible that it is mixed withaudio objects associated with different metadata. In this regard,metadata of audio objects can be preserved after clustering. Otheradvantages achieved by the example embodiments will become apparentthrough the following descriptions.

DESCRIPTION OF DRAWINGS

Through the following detailed description with reference to theaccompanying drawings, the above and other objectives, features andadvantages of embodiments will become more comprehensible. In thedrawings, several example embodiments will be illustrated in an exampleand non-limiting manner, wherein:

FIG. 1 illustrates a flowchart of a method for metadata-preserved audioobject clustering in accordance with an example embodiment;

FIG. 2 illustrates a schematic diagram for an audio object clusteringprocess in accordance with an example embodiment;

FIG. 3 illustrates a block diagram of a system for metadata-preservedaudio object clustering in accordance with an example embodiment; and

FIG. 4 illustrates a block diagram of an example computer systemsuitable for implementing embodiments.

Throughout the drawings, the same or corresponding reference symbolsrefer to the same or corresponding parts.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Principles of the example embodiment will now be described withreference to various example embodiments illustrated in the drawings. Itshould be appreciated that the depiction of these embodiments is only toenable those skilled in the art to better understand and furtherimplement the example embodiments; it is not intended for limiting thescope in any manner.

As mentioned above, due to the limit in coding/decoding rate andtransmission bandwidth, the number of audio objects used to produceadaptive audio content may be reduced by means of clustering. Besidesthe metadata describing its spatial position, an audio object usuallyhas other metadata describing its attributes, such as size, zone masks,snap, and type of content, etc., each of which describes an artisticintent about how the audio object should be processed when it isrendered. However, in some prior methods, after audio objects areclustered, only positional metadata remains. Although other metadata maybe pre-rendered to an internal channel layout, such as in a 7.1.2 or7.1.4 system, it does not work well for all systems. Especially when theaudio objects are down-mixed to, for example, a 5.1 or 7.1 system,artistic intents of the audio objects may be violated when rendered.

Take metadata “zone mask” as an example, which has multiple modes andeach mode defines a region to which an audio object should not berendered. One mode of zone mask is “no sides”, describing that sidespeakers should be masked when rendering the audio object. By utilizingthe traditional clustering method, if an audio object at a spatialposition z=1 is rendered to a 5.1 system with metadata “no sides”, theside speakers may be activated in 5.1 rendering, since the sound at theceiling speakers may be folded to the sides. This violates the originalartistic intent. In order to address this issue, the metadata “zonemask” in the clustering process should be preserved so that it can becorrectly processed in an audio renderer.

In another example, dialog objects may be expected to be separated fromother objects after clustering, which may have many benefits forsubsequent audio object processing. For example, in subsequent audioprocessing such as dialog enhancement, separated dialog object clusterscan be easily enhanced by simply applying a gain/gains. Otherwise, itmay be very difficult to separate a dialog object if it is mixed withother objects in a cluster. In the application of dialog replacement,dialog in each language may be completely separated from each other. Forthose purposes, dialog objects should be preserved and allocated intoseparated specific clusters in the clustering process.

Further, an audio object may be associated with metadata describing itsrendering mode, for example, rendering as left total/right total (Lt/Rt)or as binaural with a head related transfer function (HRTF) whenprocessing in a headphone renderer. These rendering modes are alsoexpected to be preserved after clustering in order to generate the bestrendering results.

Therefore, to achieve better audio experience, it is desired to havemetadata preserved in audio object clustering. Example embodimentsdisclosed herein propose a method and system for metadata-preservedobject clustering.

Reference is first made to FIG. 1 which depicts a flowchart of a method100 for metadata-preserved audio object clustering in accordance withexample embodiments.

At S101, a plurality of audio objects are classified into a number ofcategories based on information to be preserved in metadata associatedwith the plurality of audio objects. The audio objects are provided asan input, and there may be tens, hundreds, or sometimes thousands ofinput audio objects.

As used herein, the information to be preserved in metadata associatedwith each audio object may indicate a processing intent when the audioobject is rendered. The information may describe how the audio objectshould be processed when it is rendered. In some embodiments, theinformation may include one or more of size information, zone maskinformation, snap information, type of content, or a rendering mode ofan audio object. The size information may be used to indicate a spatialarea or volume occupied by an audio object. The zone mask informationindicates a mode of zone mask, defining a region to which an audioobject should not be rendered. For example, the zone mask informationmay indicate a mode of “no sides”, “surround only”, “front only”, or thelike. The snap information indicates whether or not an audio objectshould be directly panned to the nearest speaker.

It should be noted that some examples of information to be preserved inmetadata are described and other information included in the metadata(such non-limiting examples include spatial position, spatial width, andthe like) may also be considered in the audio object classification,according to preference of the user or audio engineer. In someembodiments, all information in metadata associated with an audio objectmay be considered.

The number of categories may be dependent on the information in metadataof the audio objects and may be larger than or equal to one. In oneembodiment, an audio object without the information to be preserved maybe classified into one category and an audio object with differentinformation to be preserved may be classified into a different category.That is, depending on different information to be preserved,corresponding audio objects are classified into different categories.Alternatively, a category may represent a unique combination ofdifferent information to be preserved in metadata. All other audioobjects without the information of interest may be contained in onecategory or multiple categories in some cases. The scope of the exampleembodiments are not limited in this regard.

The categories may be given by manual assignment, automatic assignment,or a combination thereof. For example, a user or an audio engineer maylabel audio objects associated with different kind of metadata bydifferent flags and then those labeled audio objects may be classifiedinto different categories according to their flags. For another example,the information to be preserved in metadata may be automaticallyidentified. The user or audio engineer may also preconfigure theirpreference or expectation, such as separating dialog objects, separatingdifferent dialog languages, and/or separating different modes of zonemask. According to the pre-configuration, the audio objects may beclassified to different categories.

Assume that there are O audio objects. In the classifying process, theinformation to be preserved in metadata of audio objects may be derivedfrom (1) manual labels of metadata provided by user input, such aslabels of zone mask or snap or content type or language, and/or (2)automatic identification/labeling of metadata, such as but not limitedto, identification of the content type. The number N of possiblecategories may be determined according to the derived information, eachconsisting of a unique combination of information to be preserved. Afterclassifying, each audio object may have associated categoryidentification n_(o).

With reference to FIG. 2, a schematic diagram of audio object clusteringis illustrated. As shown in FIG. 2, based on the information to bepreserved in metadata, a plurality of input audio objects are classifiedinto five categories, Categories 0 to 4. One example of the categoriesmay be given as below:

-   -   Category 0: All audio objects without the information to be        preserved;    -   Category 1: Music objects, no zone mask;    -   Category 2: Sound effect objects, with zone mask “surround        only;”    -   Category 3: English dialog objects; and    -   Category 4: Spanish dialog objects, with zone mask “front only.”

The input audio objects may include one or more frames. A frame is aprocessing unit for audio content and the duration of a frame may bevaried and may depend on the configuration of the audio processingsystem. As the audio objects to be classified may be varied fordifferent frames in time and their metadata may also be varied, thevalue of the number of categories may also change over time. Thecategories representing different kinds of information to be preservedmay be predefined by the user or by default and input audio objects inone or more frames may then be classified into the predeterminedcategories based on the information. The categories with audio objectsclassified may be considered and those without audio objects may beignored in subsequent processing. For example, when there is no audioobject without the information to be preserved in FIG. 2, thecorresponding Category 0 may be omitted. It is contemplated that thenumber of audio objects classified in each category may change overtime.

At S102, a predetermined number of clusters are assigned to thecategories. The predetermined number may be larger than one and may bedependent on the transmission bandwidth and a coding/decoding rate ofthe audio processing system. There may be a tradeoff between thetransmission bandwidth (and/or the coding rate, and/or the decodingrate) and error criterion of the output audio objects. For example, thepredetermined number may be 11 or 16. Other values also may bedetermined, such as 5, 7, or 20, and the scope of the exampleembodiments are not limited in this regard.

In some embodiments, the predetermined number may not be varied withinthe same processing system. In some other embodiments, the predeterminednumber may be varied for different audio files to be processed.

In example embodiments disclosed herein, audio objects are firstclassified into categories according to the metadata at S101, such thateach category may represent different information to be preserved or aunique combination of different information to be preserved. Then audioobjects in those categories may be clustered in subsequent processing.There may be various approaches to assign/allocate the predeterminedoverall number of clusters to the categories. In some exampleembodiments, since the overall number of clusters is predetermined andfixed, it is possible to determine the number of clusters to be assignedto each category before clustering the audio objects. Some exampleembodiments will now be discussed.

In one example embodiment, cluster assignment may be dependent on theimportance of the plurality of audio objects. In particular, thepredetermined number of audio objects from the plurality of audioobjects first may be determined based on the importance of each audioobject relative to other audio objects and then distribution of thepredetermined number of audio objects among the categories may bedetermined. The predetermined number of clusters are correspondinglyassigned to the categories according to the distribution.

The importance of each audio object may be associated with one or moreof the types of content, partial loudness level, or energy level of theaudio object. An audio object with a great importance may represent thatthe audio object is perceptually salient among the input audio objects,for example, because of its partial loudness or energy level. In someuse cases, one or more types of content may be regarded as important,and then great importance may be given to corresponding audio objects.For example, a greater importance may be assigned to dialog objects. Itshould be noted that there are many other ways to determine or definethe importance of each audio object. For example, the importance levelof some audio objects may be specified by users. The scope of theexample embodiments are not limited in this regard.

Assume that the predetermined overall cluster number is M. In a firststep, up to M audio objects that are most important among the inputaudio objects are selected. As all input audio objects are classifiedinto corresponding categories in S101, in a second step, distribution ofthe M most important audio objects among the categories may bedetermined. Based on how many of the M audio objects are distributed ina category, the equal number of clusters may be assigned to thecategory.

With reference to FIG. 2, for example, eleven of the most importantaudio objects (illustrated as circle 201) are determined from aplurality of input audio objects (illustrated as a collection of circles201 and 202). After classifying all input audio objects into the fivecategories, Categories 0 to 4, it can be seen from FIG. 2 that four mostimportant audio objects are classified into Category 0, three mostimportant audio objects are classified into Category 1, one mostimportant audio object is classified into Category 2, two most importantaudio objects are classified into Category 3, and one most importantaudio object is classified into Category 4. It results in that 4, 3, 1,2, and 1 clusters are respectively assigned to Categories 0 to 4, asshown in FIG. 2.

It should be noted that the above described example of an importancecriterion in accordance with the example embodiment of the exampleembodiments may not be that strict. That is, it is not necessary thatthe most important audio objects are selected. In some embodiments, animportance threshold may be configured. Among those audio objects whoseimportance is higher than the threshold, the predetermined number ofaudio objects may be selected randomly.

Besides the importance criterion, cluster assignment may be performedbased on reducing an overall spatial distortion for the categories. Thatis, the predetermined number of clusters may be assigned to thecategories based on reducing or even minimizing an overall spatialdistortion for the categories.

In one example embodiment, the overall spatial distortion for thecategories may include a weighted sum of individual spatial distortionsof the categories. The weight of the corresponding category mayrepresent importance of the category or importance of the information tobe preserved associated with the category. For example, a category witha greater importance may have a larger weight. In another embodiment,the overall spatial distortion for the categories may include a maximumspatial distortion among individual spatial distortions of thecategories. It should be contemplated that it is not necessary that onlythe maximum is selected, and in some embodiments, other spatialdistortions among the categories, such as the second largest spatialdistortion, the third largest spatial distortion, or the like, may beregarded as the overall spatial distortion.

The spatial distortion for each category may be represented by thedistortion level of audio objects included in the category and thedistortion level of each audio object may be measured by the differencebetween its original spatial position and its position after beingclustered. Generally the clustered position of an audio object isdependent on the spatial position of the cluster(s) where it isallocated. In this sense, the spatial distortion for each category isassociated with the original spatial position of each audio object inthe category, and the spatial position of the cluster(s). The originalspatial position of an audio object may be contained in the metadata ofthe audio object and may, for example, consist of 3 Cartesiancoordinates (or similarly for example, consisting of polar coordinatesor cylindrical and spherical coordinates, homogenous coordinates, linenumber coordinates and the like). In one embodiment, to calculate thespatial distortion for each category, a reconstructed spatial positionof each audio object in the category may be determined based on thespatial position of the cluster(s). Then, the spatial distortion foreach category may be calculated based on a distance between the originalspatial position of each audio object in the category and thereconstructed spatial position of the audio object. The reconstructedspatial position of the audio object is a spatial position of the audioobject represented by one or more corresponding spatial clusters. Oneexample approach of determination of the reconstructed spatial positionwill be described below.

The spatial distortion with regard to different cluster numbers mayfirst be calculated for each category in order to obtain the overallspatial distortion. There are many approaches to determine the spatialdistortion for a category of audio objects. One approach is given asexample below. It should be noted that other existing ways to measurespatial distortions of the audio objects (and thus of the categories)may be applied.

Assume that for a category n, there are M_(n) cluster centroids,denoting {C_(n)(1), C_(n)(2), . . . , C_(n)(M_(n))} with spatialposition {{right arrow over (p)}_(n,1), {right arrow over (p)}_(n,2), .. . , {right arrow over (p)}_(n,M) _(n) }.dis(o_(n)(i),{C_(n)(1),C_(n)(2), . . . ,C_(n)(M_(n))}) may represent thespatial distortion for the audio object o_(n)(i) when clustering it intoM_(n) cluster centroids (assuming that audio objects in one category areonly allocated into clusters associated with the category in this case).The spatial distortion for the category n may be represented as:

$\begin{matrix}{{D_{n}\left( M_{n} \right)} = {\sum\limits_{i = 1}^{O_{n}}\; {{dis}\left( {{o_{n}(i)},\left\{ {{C_{n}(1)},{C_{n}(2)},\ldots \mspace{14mu},{C_{n}\left( M_{n} \right)}} \right\}} \right)}}} & (1)\end{matrix}$

where O_(n) represents the number of audio objects in the category n,and o_(n)(i) represents the i^(th) audio object in the category n. Insome embodiments, C_(n)(m) may be the spatial position of an audioobject with the m^(th) largest importance in the category, and thespatial position of C_(n)(m) may be the spatial position of that audioobject. The spatial distortion dis(o_(n)(i),{C_(n)(1),C_(n)(2), . . .,C_(n)(M_(n))}) may be determined by a distance (or a distance squared)between the spatial position {right arrow over (p)}_(n,i) of each audioobject o_(n)(i) and a reconstructed spatial position {right arrow over(p)}_(n,i)′ of the audio object if clustered into the M_(n) clusters.

With the spatial distortion for each category obtained, in oneembodiment, the overall spatial distortion for the categories may bedetermined as a weighted sum of individual spatial distortions of thecategories as mentioned above. For example, the overall spatialdistortion may be determined as below:

$\begin{matrix}{{Cost} = {\sum\limits_{n = 0}^{N}\; \left( {a_{n}{D_{n}\left( M_{n} \right)}} \right)}} & (2)\end{matrix}$

where N represents the number of overall categories. The gain a_(n) foreach category may be predetermined and may represent the importance ofthe corresponding category or information to be preserved in metadataassociated with the category.

In another embodiment, the overall spatial distortion for the categoriesmay be determined as a maximum spatial distortion among individualspatial distortions of the categories. For example, the overall spatialdistortion may be determined as below:

Cost=max(a ₀ D ₀(M ₀),a ₁ D ₁(M ₁), . . . ,a _(n) D _(n)(M _(n)))  (3)

In this way, the number of clusters to be assigned for each categoryM_(n) may be determined based on reducing or minimizing the overallspatial distortion metric, with the constraint

${\sum\limits_{n = 0}^{N}M_{n}} = {M.}$

That is, the overall number of assigned clusters is equal to thepredetermined number M.

The input audio objects are generally in one frame of an audio signal.Due to the typical dynamic nature of the audio signal and given that theaudio object number changes in each category, the number of clustersassigned to each category may typically vary over time. Since thechanged number of clusters for each category may cause some instabilityissues, a modified spatial distortion considering the cluster numberconsistence is utilized in the cost metric. Consequently, the costmetric may be defined as a time function. In particular, the spatialdistortion for each category is further based on the difference betweenthe number of clusters assigned to the category in the current frame andthe number of clusters assigned to the category in a previous frame. Inthis regard, the overall spatial distortion in Equation (2) may bemodified as below:

$\begin{matrix}{{Cost} = {\sum\limits_{n = 0}^{N}\left( {a_{n}{f\left( {{D_{n}\left( M_{n} \right)},M_{n},M_{n}^{\prime}} \right)}} \right)}} & (4)\end{matrix}$

The overall spatial distortion in Equation (3) may be modified as below:

Cost=max(a ₀ f(D ₀(M ₀),M ₀ ,M ₀′),a ₁ f(D ₁(M ₁),M ₁ ,M ₁′), . . . ,a_(n) f(D _(n)(M _(n)),M _(n) ,M _(n)′))  (5)

In Equations (4) and (5), M_(n) represents the cluster number of thecategory n in the current frame, M_(n)′ represents the cluster number ofthe category n in the previous frame, and f(D_(n)(M_(n)),M_(n),M_(n)′)represents the modified overall spatial distortion.

If the cluster number assigned to a category changes in the currentframe, in comparison with the previous spatial distortion, the modifiedspatial distortion may be increased to prevent the change of clusternumber. In one embodiment, f(D_(n)(M_(n)),M_(n),M_(n)′) may bedetermined as below:

f(D _(n)(M _(n)),M _(n) ,M _(n)′)=D _(n)(M _(n))+β₁ *|M _(n) −M_(n)′|  (6)

where β₁ represents a parameter with a positive value. With the modifiedspatial distortion, there is a penalty for the cluster number change foreach category. Therefore, spatial instability introduced by variation ofcluster number could be alleviated.

Since a decline in the cluster number of a category is more likely tointroduce spatial instability than an increase in the cluster number, inanother embodiment, f(D_(n) (M_(n)),M_(n),M_(n)′) may be determined asbelow:

$\begin{matrix}{{f\left( {{D_{n}\left( M_{n} \right)},M_{n},{M^{\prime}}_{n}} \right)} = \left\{ \begin{matrix}\left( {D_{n}\left( M_{n} \right)} \right. & {{{if}\mspace{14mu} M_{n}} \geq M_{n}^{\prime}} \\\left( {{D_{n}\left( M_{n} \right)}*\beta_{2}} \right. & {{{if}\mspace{14mu} M_{n}} < M_{n}^{\prime}}\end{matrix} \right.} & (7)\end{matrix}$

where β₂ represents a parameter with a value larger than 1. In thisembodiment, there is a large penalty for decreasing the cluster number,especially when the spatial distortion of the category with a decreasedcluster number is large. Therefore, spatial instability introduced bythe decreased cluster number can be reduced.

In the above description, with respect to the cluster assignment basedon reducing the overall spatial distortion, a large amount ofcalculation effort may be involved in determining the optimal number ofclusters for each category. To efficiently determine the cluster numberfor each category, in one embodiment, an iterative process is proposed.That is, the optimal cluster number of each category is estimated bymaximizing the cost reduction in each iteration of cluster assignmentprocess, so that the overall spatial distortion for the categories maybe iteratively reduced or even minimized.

By iterating from 1 to the predetermined cluster number M, in eachiteration, one or more clusters are assigned to a category which needsthem most. Denote Cost(m−1) and Cost(m) as the overall spatialdistortion in the (m−1)^(th) and the m^(th) iteration. In the m^(th)iteration, one or more new clusters may be assigned to category n* whichcan reduce the overall spatial distortion most. Therefore, n* may bedetermined by enlarging or maximizing the reduction of the overallspatial distortion, which may be represented as below:

$\begin{matrix}{\overset{\_}{n^{*}} = {\max\limits_{n^{*}}\left\{ {{{Cost}\left( {m - 1} \right)} - {{Cost}(m)}} \right\}}} & (8)\end{matrix}$

The iterative process may be based on at least one of the differencebetween a spatial distortion for a category in the current iteration andin a previous iteration or an amount of a spatial distortion for acategory in a previous iteration.

For the overall spatial distortion obtained by a weighted sum of allspatial distortions of the categories, the iterative process may bebased on difference between a spatial distortion for a category in thecurrent iteration and in a previous iteration. In each iteration, atleast one cluster may be assigned to a category that has its spatialdistortion in the current iteration become sufficiently lower (accordingto a first predetermined level) than its spatial distortion in theprevious iteration if the category is assigned with the at least onecluster. In one embodiment, the at least one cluster may be assigned tothe category having the most reduced spatial distortion if it isassigned with the at least one cluster. For example, in this embodiment,n* may be determined as below:

$\begin{matrix}{\overset{\_}{n^{*}} = {\max\limits_{n^{*}}\left\{ {{D_{n^{*}}\left( M_{n^{*},{m - 1}} \right)} - {D_{n^{*}}\left( {M_{n^{*},{m - 1}} + 1} \right)}} \right\}}} & (9)\end{matrix}$

where M_(n*,m-1) and D_(n*)(M_(n*,m-1)) represents the cluster numberand the spatial distortion for the category n* after (m−1)^(th)iteration. M_(n*,m-1)+1 represents the cluster number of the category n*in the m^(th) iteration if in this iteration one new cluster isassigned/added to the category n*, and D_(n*)(M_(n*,m-1)+1) representsthe spatial distortion for the category n* in the m^(th) iteration. Itshould be noted that, in each iteration, more than one new cluster maybe assigned, and the category n* may be similarly determined.

For the overall spatial distortion determined as the maximum spatialdistortion among all categories, the iterative process may be based onan amount of a spatial distortion for a category in a previousiteration. In each iteration, at least one cluster may be assigned to acategory having a spatial distortion higher than a second predeterminedlevel in a previous iteration. In one embodiment, the at least onecluster may be assigned to the category having the largest spatialdistortion in a previous iteration. For example, in this embodiment, n*may be determined as below:

$\begin{matrix}{\overset{\_}{n^{*}} = {\max\limits_{n^{*}}\left\{ {D_{n^{*}}\left( M_{n^{*},{m - 1}} \right)} \right\}}} & (10)\end{matrix}$

Since the category with the largest spatial distortion in the previousiteration may have its spatial distortion reduced in the currentiteration (if it has one or more clusters assigned in the currentiteration), the overall spatial distortion, which is determined by thelargest spatial distortion among all categories, may also be reduced inthe current iteration.

It is noted that the determination provided in Equations (9) and (10)may be jointly used in one iterative process. For example, in oneiteration, Equation (9) may be used to assign new cluster(s) in thisiteration. In another iteration, Equation (10) may be used to assignother new cluster(s).

Two ways of cluster assignment have been described above, one based onthe importance of audio objects and the other based on reducing theoverall spatial distortion. Additionally or alternatively, user inputsmay also be used to guide the cluster assignment. As users may havedifferent requirements for different contents for different use cases,it may largely improve the flexibility of the clustering process. Insome embodiments, the cluster assignment may be further based on one ormore of: a first threshold for the number of clusters to be assigned toeach category, a second threshold for a spatial distortion for eachcategory, or the importance of each category relative to othercategories.

A first threshold may be predefined for the number of clusters to beassigned to each category. The first threshold may be a predeterminedminimum or maximum cluster number for each category. For example, theuser may specify that one category should have a certain minimum numberof clusters. In this case, during the process of assignment, at leastthe specified number of clusters should be assigned to the category. Inthe case where a maximum threshold is set, at most, the specified numberof clusters can be assigned to the category. The second threshold may beset to guarantee the spatial distortion for a category to be reduced toa reasonable level. The importance of each category may also bespecified by the user, or may be determined based on the importance ofaudio objects classified in the category.

In some cases, the spatial distortion for a category may be high afterthe cluster assignment is done, which may introduce an audible artifact.In order to address this issue, in some embodiments, at least one audioobject in a category may be reclassified into another category based ona spatial distortion for the category. In an example embodiment, if aspatial distortion of one of the categories is higher than apredetermined threshold, some audio objects in that category may bereclassified to another category, until the spatial distortion is lessthan (or equal to) the threshold. In some examples, audio objects may bereclassified to the category containing audio objects withoutinformation to be preserved in metadata, such as Category 0 in FIG. 2.In some embodiments where the cluster assignment is based on minimizingthe overall spatial distortion in an iterative process, the objectreallocation may also be an iterative process in which the audio objecthaving the largest spatial distortion dis(o_(n)(i),{C_(n)(1), C_(n)(2),. . . , C_(n)(M_(n))}) in each iteration may be reclassified until thecriterion of spatial distortion for the category is satisfied.

Due to the typical dynamic nature of audio signals, the importance orspatial position (and thus the spatial distortion) of audio objectschanges over time. Consequently, the cluster assignment may be timevariant, and then the number of clusters allocated to each category maybe varied over time. In this sense, the category identificationassociated with a cluster m may change over time. In particular, thecluster m may represent a certain language (for example, Spanish) duringa first frame, while it may change the category identification andconsequently the language for a second frame (for example, English).This is in contrast with legacy, channel-based systems in whichlanguages are statically coupled to channels rather than changingdynamically.

The cluster assignment at S102 is described above.

Referring back to FIG. 1, at S103, an audio object in each of thecategories is allocated to at least one of the clusters according to theassignment.

In following description, two approaches are provided for clusteringaudio objects after the audio objects are classified into the categoriesat S101 and clusters are assigned to each category at S102.

In one approach, an audio object in each category may be allocated to atleast one of the clusters assigned to one or more of the categoriesbased on reducing a distortion cost associated with the categories. Thatis, due to the limit in the number of clusters assigned for eachcategory, some leakage across clusters and categories is allowed inorder to reduce the distortion cost and avoid artifacts for complexaudio content. This approach may be referred to as a fuzzy categoryclustering. In this fuzzy category clustering approach, an audio objectmay be divided softly with a gain to different clusters in differentcategories and with a corresponding cost. During the clustering process,the distortion cost is expected to be minimal with respect to theoverall spatial distortion as well as the disadvantage or mismatch ofallocating an object in a category to a cluster of a different category.Therefore, there is a tradeoff between the cluster budget and thecomplexity of the audio content. The fuzzy category clustering approachmay be suitable for audio objects with metadata such as zone mask andsnap, since no strict separation with other metadata is required forthem. The fuzzy category clustering approach may be described in themanner set forth below.

In the fuzzy category clustering approach, the cluster number assignedto each category may be determined at S102 based on the importance ofaudio object or based on minimizing the overall spatial distortion. Forthe importance-based cluster assignment, there may be some categorieswithout any cluster assigned. In these cases, the fuzzy categoryclustering approach may be applied when clustering audio objects sincean object may be softly clustered into a cluster/clusters of othercategories. It should be noted that there may not be a necessarycorrelation between the approaches applied in the step of clusterassignment and the approaches applied in the step of audio objectclustering.

In the fuzzy category clustering approach, the distortion cost may berepresented as a cost function which is associated with one or more of:(1) an original spatial position of each audio object {right arrow over(p)}_(o), (2) identification of a category n_(o) to which each audioobject is classified, (3) a spatial position of each cluster {rightarrow over (p)}_(m), or more specifically, the spatial position of thecluster(s) to which the audio object will be allocated, or (4)identification of a category n_(m) associated with each cluster. In oneexample, the clustered audio object of a cluster may be determined byall input audio objects distributed over it using a gain g_(o,m), whichmay be represented as below:

$\begin{matrix}{y_{m} = {\sum\limits_{o = 1}^{o = O}\; {g_{o,m}x_{o}}}} & (11)\end{matrix}$

where O represents the number of input audio objects, y_(m) representsthe clustered audio object of the m^(th) cluster, x_(o) represents theo^(th) input audio object, and the gain g_(o,m) may be represented asg_(o,m)=F({right arrow over (p)}_(o),n_(o),{right arrow over(p)}_(m),n_(m)). For example, as shown in FIG. 2, an audio object inCategory 1 may be clustered into all eleven clusters with correspondinggains, regardless of categories to which the clusters are assigned.

In some embodiments, the gain g_(o,m) may be determined by minimizingthe cost function which is associated with one or more of {right arrowover (p)}_(o), n_(o), {right arrow over (p)}_(m), or n_(m). The costfunction may be based on a distance between the original spatialposition {right arrow over (p)}_(o) of each audio object and a spatialposition of a cluster {right arrow over (p)}_(m) to which the audioobject is allocated. {right arrow over (p)}_(m), as discussed above, maybe determined as the spatial position of the audio object with thegreatest importance within the m^(th) category. For example, it isdesired that the distance between {right arrow over (p)}_(o) and {rightarrow over (p)}_(m) is as small as possible. Alternatively oradditionally, the cost function may also be associated with a mismatchbetween the identification of a category n_(o) to which each audioobject is classified and the identification of a category n_(m)associated with a cluster to which the audio object is allocated.Generally, an audio object is desired to be clustered within the samecategory, and then the cost may be small.

In some embodiments, the cost function may be represented as cumulativecontributions using second-order polynomials in {right arrow over(p)}_(o), n_(o), {right arrow over (p)}_(m), and n_(m), and then theglobal minimum value may be determined from the cost function as thegain g_(o,m). The detailed discussion may be provided in the manner setforth below.

The cost function may be typically minimized subject to a certainadditional criterion. In allocating audio signals, one criterion may beto maintain the summed amplitude or energy of an input audio object, forexample,

$\begin{matrix}{{\forall_{o\; \in O}{\sum\limits_{m = 1}^{M}\; \left( g_{o,m} \right)^{\alpha}}} = 1} & (12)\end{matrix}$

where α may be a value between 1 and 2. For any audio object o, the gaing_(o,m) corresponding all M clusters may be subject to the aboveequation.

In the following, the cost function E may be discussed. By minimizingthe cost function, the gain g_(o,m), may be determined.

The cost function, as mentioned above, may be associated with thedistance between {right arrow over (p)}_(o) and {right arrow over(p)}_(m), which may be regarded as a first term E_(D) in the costfunction, and may be determined as:

$\begin{matrix}{E_{D} = {\sum\limits_{m}\; {g_{o,m}^{2}{{{\overset{\rightarrow}{p}}_{m} - {\overset{\rightarrow}{p}}_{o}}}^{2}}}} & (13)\end{matrix}$

The cost function may also be associated with a mismatch between n_(o)and n_(m), which may be regarded as a second term E_(C) in the costfunction. E_(C) may represent the cost of clustering an audio objectacross a cluster within a different category and may be determined as:

$\begin{matrix}{E_{C} = {\sum\limits_{m}\; {g_{o,m}^{2}\left( {n_{m}!=n_{0}} \right)}}} & (14)\end{matrix}$

where n_(m)!=n₀ may be determined as

$\begin{matrix}{\overset{\_}{\delta} = \left\{ \begin{matrix}{0,} & {{{if}\mspace{14mu} n_{m}} = n_{o}} \\{1,} & {{{if}\mspace{14mu} n_{m}} \neq n_{o}}\end{matrix} \right.} & (15)\end{matrix}$

As mentioned above, when minimizing the cost function, one criterion isto maintain the summed amplitude or energy of an input audio object.Therefore, the cost function may also be associated with the gain orloss of energy; that is, a deviation from the sum of gains for aspecific audio object and +1. The deviation may be regarded as a thirdterm E_(N) in the cost function, which may be determined as:

$\begin{matrix}{E_{N} = \left( {1 - {\sum\limits_{m}\; g_{o,m}^{2}}} \right)^{2}} & (16)\end{matrix}$

Furthermore, the cost function may be based on a distance between theoriginal spatial position of each audio object {right arrow over(p)}_(o) and a reconstructed spatial position of the audio object {rightarrow over (p)}_(o)′. The reconstructed spatial position {right arrowover (p)}_(o)′ may be determined according to the spatial position of acluster {right arrow over (p)}_(m) to which the audio object isclustered with the gain g_(o,m). For example, {right arrow over(p)}_(o)′ may be determined as below:

$\begin{matrix}{{\overset{\rightarrow}{p}}_{o}^{\; \prime} = {\sum\limits_{m}\; {g_{o,m}{\overset{\rightarrow}{p}}_{m}}}} & (17)\end{matrix}$

The distance between {right arrow over (p)}_(o) and {right arrow over(p)}_(o)′ may be regarded as the fourth term E_(P) in the cost functionand may be represented as below:

$\begin{matrix}{E_{P} = {{{{\overset{\rightarrow}{p}}_{o}^{\; \prime} - {{\overset{\rightarrow}{p}}_{o}{\sum\limits_{m}\; g_{o,m}}}}}^{2} = {{{\sum\limits_{m}\; {g_{o,m}{\overset{\rightarrow}{p}}_{m}}} - {{\overset{\rightarrow}{p}}_{o}{\sum\limits_{m}\; g_{o,m}}}}}^{2}}} & (18)\end{matrix}$

According to the first, second, third, and fourth terms, the costfunction may be represented as a weighted sum of those terms and may berepresented as below:

E=w _(D) E _(D) +w _(C) E _(C) +w _(N) E _(N) +w _(P) E _(P)  (19)

where the weights w_(D), w_(C), w_(N), and w_(P) may represent theimportance of different terms in the cost function.

Based on the four terms in the cost function, the gain g_(o,m) may bedetermined. An example of calculation for the gain g_(o,m) is givenbelow. It should be noted that, other methods of calculation are alsopossible.

The gain g_(o,m) of the o^(th) audio object for M clusters may bewritten as a vector:

$\begin{matrix}{{\overset{\rightarrow}{g}}_{o} = \begin{bmatrix}g_{o,1} \\M \\g_{o,M}\end{bmatrix}} & (20)\end{matrix}$

The spatial positions of the M clusters may be written as a matrix:

$\begin{matrix}{{\overset{\rightarrow}{P}}_{M} = \begin{bmatrix}{{\overset{\rightarrow}{p}}_{1}\;} \\M \\{{\overset{\rightarrow}{p}}_{M}\;}\end{bmatrix}} & (21)\end{matrix}$

A matrix for the original spatial positions of the audio object may alsobe constructed as:

$\begin{matrix}{P_{O} = \begin{bmatrix}{{\overset{\rightarrow}{p}}_{o}\;} \\M \\{{\overset{\rightarrow}{p}}_{o}\;}\end{bmatrix}} & (22)\end{matrix}$

The first term E_(D) representing the distance between the originalspatial position and the reconstructed spatial position of an audioobject may be reformulated as below:

$\begin{matrix}{E_{D} = {{\sum\limits_{m}\; {g_{o,m}^{2}{{{\overset{\rightarrow}{p}}_{m} - {\overset{\rightarrow}{p}}_{o}}}^{2}}} = {{\overset{\rightarrow}{g}}_{o}^{\; T}\Lambda_{D}{\overset{\rightarrow}{g}}_{o}}}} & (23)\end{matrix}$

where Λ_(D) represents a diagonal matrix with diagonal elementsλ_(mm)={right arrow over (p)}_(m)−{right arrow over (p)}_(o) ².

The second term E_(C) representing a mismatch between n_(o) and n_(m) ofan audio object may be reformulated as below:

$\begin{matrix}{E_{C} = {{\sum\limits_{m}\; {g_{o,m}^{2}\left( {n_{m}!=n_{o}} \right)}} = {{\overset{\rightarrow}{g}}_{o}^{\; T}\Lambda_{C}{\overset{\rightarrow}{g}}_{o}}}} & (24)\end{matrix}$

where Λ_(C) represents a diagonal matrix with diagonal elementsλ_(mm)=(n_(m)!=n_(o)).

The third term E_(N) representing the deviation the sum of gains for anaudio object and +1 may be reformulated as below:

$\begin{matrix}{E_{N} = {\left( {1 - {\sum\limits_{m}\; g_{o,m}^{2}}} \right)^{2} = {1 - {2\; J_{1,M}{\overset{\rightarrow}{g}}_{o}} + {{\overset{\rightarrow}{g}}_{o}^{T}J_{N,M}{\overset{\rightarrow}{g}}_{o}^{\;}}}}} & (25)\end{matrix}$

where J_(N,M) represents all-ones matrix with dimensions (N, M).

The fourth term E_(P) representing the distance between the originalspatial position and the reconstructed spatial position of an audioobject may be reformulated as below:

$\begin{matrix}{E_{P} = {{{{\sum\limits_{m}\; {g_{o,m}{\overset{\rightarrow}{p}}_{m}}} - {\sum\limits_{m}\; {g_{o,m}{\overset{\rightarrow}{p}}_{o}}}}}^{2} = {{{{{\overset{\rightarrow}{g}}_{o}^{T}P_{M}} - {{\overset{\rightarrow}{g}}_{o}^{T}P_{O}}}}^{2} = {\left( {{{\overset{\rightarrow}{g}}_{o}^{T}P_{M}} - {{\overset{\rightarrow}{g}}_{o}^{T}P_{O}}} \right)\left( {{{\overset{\rightarrow}{g}}_{o}^{T}P_{M}} - {{\overset{\rightarrow}{g}}_{o}^{T}P_{O}}} \right)^{T}}}}} & (26)\end{matrix}$

By combining the above Equations (23)-(26) together, the cost functionmay be represented as set forth below:

E={right arrow over (g)} _(o) ^(T) A{right arrow over (g)} _(o) +B{rightarrow over (g)} _(o) +C  (27)

with

A=w _(P)(P _(M) P _(M) ^(T) −P _(M) P _(O) ^(T) −P _(O) P _(M) ^(T) +P_(O) P _(O) ^(T))+w _(D)Λ_(D) +w _(N) J _(M,M) +w _(C)Λ_(C)  (28)

B=−2w _(N) J _(1,M)  (29)

C=w _(N)  (30)

As discussed above, it is desirable to obtain a minimum in the costfunction, which may be determined by:

$\begin{matrix}{{\frac{\partial}{\partial\overset{\rightarrow}{g}}E} = 0} & (31)\end{matrix}$

giving

(A+A ^(T)){right arrow over (g)} _(o) +B ^(T)=0  (32)

Finally, the vector

g o

may be determined as below:

{right arrow over (g)} _(o)=−(A+A ^(T))⁻¹ B ^(T)  (33)

By calculating the above equation, the gains for the o^(th) audio objectamong the M clusters may be determined.

The o^(th) audio object may be clustered into M clusters with thedetermined gain vector {right arrow over (g)}_(o). It can be appreciatedthat, depending on the determined gain vector, an audio object may beclustered into only one cluster of one category where it is classifiedor of a different category, or may be clustered in to multiple clustersof one category where it is classified or of multiple differentcategories.

The reconstructed spatial position of an audio object may be obtained bythe Equation (17) when the gain vector {right arrow over (g)}_(o) isdetermined. In this regard, the process of determining the gains mayalso be applied in the cluster assignment based on minimizing theoverall spatial distortion as described above so as to determine thereconstructed spatial position and thus the spatial position of eachcategory.

It should be noted that the second-order polynomial is used as anexample to determine the minimum in the cost function. Many otherexponential values, for example, 1, 1.5, 3, and the like may also beused in other example embodiments.

The fuzzy category clustering approach for audio object clustering isdescribed above. In another approach, an audio object in each categorymay be allocated to at least one of clusters that are assigned to thecategory, based on reducing a spatial distortion cost associated withthe category. That is, no leakage across categories is allowed. Theaudio object clustering is performed within each category and an audioobject may not be grouped into a cluster assigned to another category.This approach may be referred to as a hard category clustering approach.In some embodiments where the approach is applied, an audio object maybe allocated to more than one of the clusters assigned to the categorycorresponding to the audio object. In a further embodiment, no leakageacross clusters is allowed in audio object clustering and an audioobject may be allocated to only one of the clusters assigned to thecorresponding category.

The hard category clustering approach may be suitable for some specificapplications, such as dialog replacement or dialog enhancement, whichrequire the audio objects (dialog objects) to be separated with others.

In the hard category clustering approach, since an audio object in onecategory may not be clustered into one or more clusters of othercategories, it is expected that in the previous cluster assignment, atleast one cluster is assigned to each category. For this purpose, thecluster assignment by minimizing the overall spatial distortiondescribed above may be more suitable in some embodiments. In otherembodiments, the importance-based cluster assignment may also be usedwhen hard category clustering is applied. Some additional conditions maybe used in the cluster assignment to ensure that each category has atleast one cluster assigned, as discussed above. For example, a minimumthreshold of cluster or a minimum threshold of spatial distortion foreach category may be utilized.

Within the category, the audio object, in one or more exampleembodiments may be clustered in only one cluster or in multiple clusterssince the category represents the same kind of metadata. For example, asshown in FIG. 2, an audio object in Category 1 may be clustered into oneor more of Clusters 4, 5, or 6. In a scenario where an audio object isclustered into multiple clusters within one category, correspondinggains may also be determined to reduce or even minimize the distortioncost associated with the category (which may be similar to what isdescribed with respect to the fuzzy category clustering approach). Thedifference lies in that the determination is performed within onecategory. In some embodiments, each input audio object may be allowed tobe clustered to only one cluster assigned to its category.

Two approaches for audio clustering are discussed above. It should benoted that the two approaches may be utilized separately or incombination. For example, after audio object classification at S101 andcluster assignment at S102, for some of the categories, the fuzzycategory clustering approach may be applied to cluster audio objectswithin them; and for the remaining categories, the hard categoryclustering approach may be applied. That is, some leakage acrosscategories may be allowable within some categories and no leakage acrosscategories is allowable for other categories.

After input audio objects are allocated to the clusters, for eachcluster, audio objects may be combined to obtain a clustered audioobject, and the metadata of audio objects in each cluster may becombined to obtain the metadata of the clustered audio object. Theclustered audio object may be a weighed sum of all audio objects in thecluster with corresponding gains. The metadata of the clustered audioobject may be the corresponding metadata representing by the category insome examples, or may be metadata of any audio object or the mostimportant audio object among the cluster or its category in otherexamples.

Since all input audio objects are classified into correspondingcategories depending on their information to be preserved in metadatabefore audio object clustering, different metadata to be preserved or aunique combination of metadata to be preserved is associated with adifferent category. After clustering, for an audio object within onecategory, it is less possible that it is mixed with audio objectsassociated with different metadata. In this regard, the metadata of anaudio object can be preserved after clustering. Furthermore, during thecluster assignment and audio object allocation process, the spatialdistortion or distortion cost is considered.

FIG. 3 depicts a block diagram of a system 300 for metadata-preservedaudio object clustering in accordance with one example embodiment. Asdepicted in FIG. 3, the system 300 comprises an audio objectclassification unit 301 configured to classify a plurality of audioobjects into a number of categories based on information to be preservedin metadata associated with the plurality of audio objects. The system300 further comprises a cluster assignment unit 302 configured to assigna predetermined number of clusters to the categories, and an audioobject allocation unit 303 configured to allocate an audio object ineach of the categories to at least one of the clusters according to theassigning.

In some embodiments, the information may include one or more of sizeinformation, zone mask information, snap information, type of content,or a rendering mode of an audio object.

In some embodiments, the audio object classification unit 301 may befurther configured to classify an audio object without the informationto be preserved into one category; and classify an audio object withdifferent information to be preserved into a different category.

In some embodiments, the cluster assignment unit 302 may furthercomprise: an importance based determination unit configured to determinethe predetermined number of audio objects from the plurality of audioobjects based on an importance of each audio object relative to otheraudio objects; and a distribution determination unit configured todetermine distribution of the predetermined number of audio objectsamong the categories. In these embodiments, the cluster assignment unit302 may be further configured to assign the predetermined number ofclusters to the categories according to the distribution.

In some embodiments, the cluster assignment unit 302 may be furtherconfigured to assign the predetermined number of clusters to thecategories based on reducing an overall spatial distortion for thecategories.

In some embodiments, the overall spatial distortion for the categoriesmay include a maximum spatial distortion among individual spatialdistortions of the categories, or a weighted sum of individual spatialdistortions of the categories. A spatial distortion for each categorymay be associated with an original spatial position of each audio objectin the category and a spatial position of at least one of the clusters.

In some embodiments, a reconstructed spatial position of each audioobject may be determined based on the spatial position of the at leastone cluster, and the spatial distortion for each category may bedetermined based on a distance between the original spatial position ofeach audio object in the category and the reconstructed spatial positionof the audio object.

In some embodiments, the plurality of audio objects may be in one frameof an audio signal, and a spatial distortion for each category may befurther based on difference between the number of clusters assigned tothe category in current frame and the number of clusters assigned to thecategory in a previous frame.

In some embodiments, the cluster assignment unit 302 may be furtherconfigured to iteratively reduce the overall spatial distortion for thecategories based on at least one of the following: an amount of aspatial distortion for a category in a previous iteration, or differencebetween a spatial distortion for a category in current iteration and ina previous iteration.

In some embodiments, the cluster assignment unit 302 may be furtherconfigured to assign the predetermined number of clusters to thecategories based on one or more of the following: a first threshold forthe number of clusters to be assigned to each category, a secondthreshold for a spatial distortion for each category, or an importanceof each category relative to other categories.

In some embodiments, the system 300 may further comprise an audio objectreclassification unit configured to reclassify at least one audio objectin a category into another category based on a spatial distortion forthe category.

In some embodiments, the audio object allocation unit 303 may be furtherconfigured to allocate an audio object in each category to at least oneof the clusters assigned to the category based on reducing a distortioncost associated with the category.

In some embodiments, the audio object allocation unit 303 may be furtherconfigured to allocate an audio object in each category to at least oneof the clusters assigned to one or more of the categories based onreducing a distortion cost associated with the categories.

In some embodiments, the distortion cost may be associated with one ormore of an original spatial position of each audio object, a spatialposition of the at least one cluster, identification of a category towhich each audio object is classified, or identification of eachcategory to which the at least one cluster is assigned.

In some embodiments, the distortion cost may be determined based on oneor more of the following: a distance between the original spatialposition of each audio object and the spatial position of the at leastone cluster, a distance between the original spatial position of eachaudio object and a reconstructed spatial position of the audio objectdetermined based on the spatial position of the at least one cluster, ora mismatch between the identification of the category to which eachaudio object is classified and the identification of each category towhich the at least one cluster is assigned.

In some embodiments, the system 300 may further comprise an audio objectcombining unit configure to combine audio objects in each cluster toobtain a clustered audio object and a metadata combining unit configureto combine metadata of audio objects in each cluster to obtain metadataof the clustered audio object.

For the sake of clarity, some additional components of the system 300are not depicted in FIG. 3. However, it should be appreciated that thefeatures as described above with reference to FIG. 1 are all applicableto the system 300. Moreover, the components of the system 300 may be ahardware module or a software unit module and the like. For example, insome embodiments, the system 300 may be implemented partially orcompletely with software and/or firmware, for example, implemented as acomputer program product embodied in a computer readable medium.Alternatively or additionally, the system 300 may be implementedpartially or completely based on hardware, for example, as an integratedcircuit (IC), an application-specific integrated circuit (ASIC), asystem on chip (SOC), a field programmable gate array (FPGA), and soforth. The scope of the example embodiments are not limited in thisregard.

FIG. 4 depicts a block diagram of an example computer system 400suitable for implementing embodiments. As shown, the computer system 400comprises a central processing unit (CPU) 401 which is capable ofperforming various processes in accordance with a program stored in aread only memory (ROM) 402 or a program loaded from a storage section408 to a random access memory (RAM) 403. In the RAM 403, the datarequired when the CPU 401 performs the various processes or the like isalso stored as required. The CPU 401, the ROM 402 and the RAM 403 areconnected to one another via a bus 404. An input/output (I/O) interface405 is also connected to the bus 404.

The following components are connected to the I/O interface 405: aninput section 406 including a keyboard, a mouse, or the like; an outputsection 407 including a display such as a cathode ray tube (CRT), aliquid crystal display (LCD), or the like, and a loudspeaker or thelike; the storage section 408 including a hard disk or the like; and acommunication section 409 including a network interface card such as aLAN card, a modem, or the like. The communication section 409 performs acommunication process via the network such as the internet. A drive 410is also connected to the I/O interface 405 as required. A removablemedium 411, such as a magnetic disk, an optical disk, a magneto-opticaldisk, a semiconductor memory, or the like, is mounted on the drive 410as required, so that a computer program read therefrom is installed intothe storage section 408 as required.

Specifically, in accordance with example embodiments disclosed herein,the processes described above with reference to FIG. 1 may beimplemented as computer software programs. For example, embodiments ofthe example embodiments include a computer program product including acomputer program tangibly embodied on a machine readable medium, thecomputer program including program code for performing method 100. Insuch embodiments, the computer program may be downloaded and mountedfrom the network via the communication section 409, and/or installedfrom the removable medium 411.

Generally speaking, various example embodiments may be implemented inhardware or special purpose circuits, software, logic or any combinationthereof. Some aspects may be implemented in hardware, while otheraspects may be implemented in firmware or software which may be executedby a controller, microprocessor or other computing device. While variousaspects of the example embodiments are illustrated and described asblock diagrams, flowcharts, or using some other pictorialrepresentation, it will be appreciated that the blocks, apparatus,systems, techniques or methods described herein may be implemented in,as non-limiting examples, hardware, software, firmware, special purposecircuits or logic, general purpose hardware or controller or othercomputing devices, or some combination thereof.

Additionally, various blocks shown in the flowcharts may be viewed asmethod steps, and/or as operations that result from the operation ofcomputer program code, and/or as a plurality of coupled logic circuitelements constructed to carry out the associated function(s). Forexample, embodiments may include a computer program product comprising acomputer program tangibly embodied on a machine readable medium, thecomputer program containing program codes configured to carry out themethods as described above.

In the context of the disclosure, a machine readable medium may be anytangible medium that can contain, or store a program for use by or inconnection with an instruction execution system, apparatus, or device.The machine readable medium may be a machine readable signal medium or amachine readable storage medium. A machine readable medium may include,but not limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, or device, or any suitablecombination of the foregoing. More specific examples of the machinereadable storage medium would include an electrical connection havingone or more wires, a portable computer diskette, a hard disk, a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), an optical fiber, a portablecompact disc read-only memory (CD-ROM), an optical storage device, amagnetic storage device, or any suitable combination of the foregoing.

Computer program code for carrying out methods of the exampleembodiments may be written in any combination of one or more programminglanguages. These computer program codes may be provided to a processorof a general purpose computer, special purpose computer, or otherprogrammable data processing apparatus, such that the program codes,when executed by the processor of the computer or other programmabledata processing apparatus, cause the functions/operations specified inthe flowcharts and/or block diagrams to be implemented. The program codemay be executed entirely on a computer, partly on the computer, as astand-alone software package, partly on the computer and partly on aremote computer or entirely on the remote computer or server. Theprogram code may be distributed on specially-programmed devices whichmay be generally referred to herein as “modules”. Software componentportions of the modules may be written in any computer language and maybe a portion of a monolithic code base, or may be developed in morediscrete code portions, such as is typical in object-oriented computerlanguages. In addition, the modules may be distributed across aplurality of computer platforms, servers, terminals, mobile devices andthe like. A given module may even be implemented such that the describedfunctions are performed by separate processors and/or computing hardwareplatforms.

As used in this application, the term “circuitry” refers to all of thefollowing: (a) hardware-only circuit implementations (such asimplementations in only analog and/or digital circuitry) and (b) tocombinations of circuits and software (and/or firmware), such as (asapplicable): (i) to a combination of processor(s) or (ii) to portions ofprocessor(s)/software (including digital signal processor(s)), software,and memory(ies) that work together to cause an apparatus, such as amobile phone or server, to perform various functions) and (c) tocircuits, such as a microprocessor(s) or a portion of amicroprocessor(s), that require software or firmware for operation, evenif the software or firmware is not physically present. Further, it iswell known to the skilled person that communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia.

Further, while operations are depicted in a particular order, thisshould not be understood as requiring that such operations be performedin the particular order shown or in sequential order, or that allillustrated operations be performed, to achieve desirable results. Incertain circumstances, multitasking and parallel processing may beadvantageous. Likewise, while several specific implementation detailsare contained in the above discussions, these should not be construed aslimitations on the scope of any what may be claimed, but rather asdescriptions of features that may be specific to particular exampleembodiments. Certain features that are described in this specificationin the context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesub-combination.

Various modifications and adaptations to the foregoing exampleembodiments of may become apparent to those skilled in the relevant artsin view of the foregoing description, when it is read in conjunctionwith the accompanying drawings. Any and all modifications will stillfall within the scope of the non-limiting and example embodiments.Furthermore, other example embodiments set forth herein will come tomind to one skilled in the art to which these embodiments pertain havingthe benefit of the teachings presented in the foregoing descriptions andthe drawings.

Accordingly, the example embodiments disclosed herein may be embodied inany of the forms described herein. For example, the following enumeratedexample embodiments (EEEs) describe some structures, features, andfunctionalities of some aspects of the example embodiments disclosedherein.

EEE 1. A method to preserve object metadata in audio object clustering,including: allocating audio objects into categories, with each categoryrepresenting one or a unique combination of metadata that requirespreservation; generating a number of clusters for each category througha clustering process, subject to an overall (maximum) number ofavailable clusters and an overall error criterion, and the methodfurther comprises: a fuzzy object category separation, or a hard objectcategory separation.

EEE 2. The method according to EEE 1, wherein the fuzzy object categoryseparation comprises: determining output cluster centroids, for example,by selecting the most important objects, and generating the outputcluster signals, by minimizing a cost function which jointly considers(1) the positional metadata of each object {right arrow over (p)}_(o),(2) the category identification of each object n_(o), (3) the positionalmetadata of each cluster {right arrow over (p)}_(m), and (4) thecategory identification associated with each cluster n_(m).

EEE 3. The method according to EEE 2, wherein the cost functionconsiders a cost associated with a mismatch between the object categoryidentification n_(o) and the cluster category identification n_(m);

EEE 4. The method according to EEE 1, wherein the hard object categoryseparation comprises: determining an optimal cluster number for eachcategory by minimizing the overall spatial distortion, and clusteringobjects within each category, the clustering process is performed foreach category independently.

EEE 5. The method according to EEE 4,wherein the overall spatialdistortion comprises: a spatial distortion in each category measuringdifference between the original object position and the position afterclustering, the importance of each category, and the cluster numberchange of each category.

EEE 6. The method according to EEE 4, the process of determining theoptimal cluster number for each category is an iterative process, and acluster is added or assigned to the category which needs it most in eachiteration.

EEE 7. The method according to EEE 4, the process of determining theoptimal cluster number further comprises object reallocation in order toavoid large spatial distortion in one category.

It will be appreciated that the embodiments of the example embodimentsdisclosed herein are not to be limited to the specific embodimentsdisclosed and that modifications and other embodiments are intended tobe included within the scope of the appended claims. Although specificterms are used herein, they are used in a generic and descriptive senseonly, and not for purposes of limitation.

What is claimed is: 1-33. (canceled)
 34. A method for metadata-preservedaudio object clustering, comprising: classifying a plurality of audioobjects into a number of categories based on information to be preservedin metadata associated with the plurality of audio objects; determining,for each category, a respective number of clusters to be assigned to therespective category, so that the overall number of clusters adds up tothe predetermined number of clusters; and allocating an audio object ineach of the categories to at least one of the clusters according to theassigning.
 35. The method according to claim 34, wherein the informationincludes one or more of size information, zone mask information, snapinformation, type of content, or a rendering mode of an audio object.36. The method according to claim 34, wherein classifying a plurality ofaudio objects into a number of categories based on information to bepreserved in metadata associated with the plurality of audio objectscomprises: classifying an audio object without the information to bepreserved into one category; and classifying an audio object withdifferent information to be preserved into a different category.
 37. Themethod according to claim 34, wherein assigning a predetermined numberof clusters to the categories comprises: determining the predeterminednumber of audio objects from the plurality of audio objects based on animportance of each audio object relative to other audio objects;determining distribution of the predetermined number of audio objectsamong the categories; and assigning the predetermined number of clustersto the categories according to the distribution.
 38. The methodaccording to claim 34, wherein assigning a predetermined number ofclusters to the categories comprises: assigning the predetermined numberof clusters to the categories based on reducing an overall spatialdistortion for the categories.
 39. The method according to claim 38,wherein the overall spatial distortion for the categories includes amaximum spatial distortion among individual spatial distortions of thecategories, or a weighted sum of individual spatial distortions of thecategories, and wherein a spatial distortion for each category isassociated with an original spatial position of each audio object in thecategory and a spatial position of at least one of the clusters.
 40. Themethod according to claim 39, wherein a reconstructed spatial positionof each audio object is determined based on the spatial position of theat least one cluster, and the spatial distortion for each category isdetermined based on a distance between the original spatial position ofeach audio object in the category and the reconstructed spatial positionof the audio object.
 41. The method according to claim 39, wherein theplurality of audio objects are in one frame of an audio signal, and aspatial distortion for each category is further based on differencebetween the number of clusters assigned to the category in current frameand in a previous frame.
 42. The method according to claim 38, whereinassigning the predetermined number of clusters to the categories basedon reducing an overall spatial distortion for the categories comprises:iteratively reducing the overall spatial distortion for the categoriesbased on at least one of the following: an amount of a spatialdistortion for a category in a previous iteration, or difference betweena spatial distortion for a category in current iteration and in aprevious iteration.
 43. The method according to claim 37, whereinassigning the predetermined number of clusters to the categories isfurther based on one or more of the following: a first threshold for thenumber of clusters to be assigned to each category, a second thresholdfor a spatial distortion for each category, or an importance of eachcategory relative to other categories.
 44. The method according to claim34, further comprising: reclassifying at least one audio object in acategory into another category based on a spatial distortion for thecategory.
 45. The method according to claim 34, wherein allocating anaudio object in each of the categories to at least one of the clustersaccording to the assigning comprising: allocating an audio object ineach category to at least one of the clusters assigned to the categorybased on reducing a distortion cost associated with the category. 46.The method according to claim 34, wherein allocating an audio object ineach of the categories to at least one of the clusters according to theassigning comprising: allocating an audio object in each category to atleast one of the clusters assigned to one or more of the categoriesbased on reducing a distortion cost associated with the categories. 47.The method according to claim 45, wherein the distortion cost isassociated with one or more of an original spatial position of eachaudio object, a spatial position of the at least one cluster,identification of a category to which each audio object is classified,or identification of each category to which the at least one cluster isassigned.
 48. The method according to claim 47, wherein the distortioncost is determined based on one or more of the following: a distancebetween the original spatial position of each audio object and thespatial position of the at least one cluster, a distance between theoriginal spatial position of each audio object and a reconstructedspatial position of the audio object determined based on the spatialposition of the at least one cluster, or a mismatch between theidentification of the category to which each audio object is classifiedand the identification of each category to which the at least onecluster is assigned.
 49. The method according to claim 34, furthercomprising: combining audio objects in each cluster to obtain aclustered audio object; and combining metadata of audio objects in eachcluster to obtain metadata of the clustered audio object.
 50. A systemfor metadata-preserved audio object clustering, comprising: an audioobject classification unit configured to classify a plurality of audioobjects into a number of categories based on information to be preservedin metadata associated with the plurality of audio objects; a clusterassignment unit configured to determining, for each category, arespective number of clusters to be assigned to the respective category,so that the overall number of clusters adds up to the predeterminednumber of clusters; and an audio object allocation unit configured toallocate an audio object in each of the categories to at least one ofthe clusters according to the assigning.
 51. The system according toclaim 50, wherein the information includes one or more of sizeinformation, zone mask information, snap information, type of content,or a rendering mode of an audio object.
 52. The system according toclaim 50, wherein the audio object classification unit is furtherconfigured to classify an audio object without the information to bepreserved into one category, and classify an audio object with differentinformation to be preserved into a different category.
 53. A computerprogram product, comprising a computer program tangibly embodied on amachine readable medium, the computer program containing program codefor performing the method according to claim 34.